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(54) Agent interface device 

(57) An agent interface device for providing informa- 
tion suited tor user preferences on the basis of the user 
usage history. The agent interface device comprises an 
agent display control means (109) for controlling the dis- 
play and action of the agent, a voice output means (11 0) 
for generating a speech of the agent, an input means 
(111) for inputting information from the user, an action 
script execution engine (1 01 ) for driving the agent inter- 
face into action according to an action script describing 
means to control the agent interface, a retrieval engine 
(1 02) for retrieving an adapted action script from an ac- 
tion script database DB (103) and transferring it to the 
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action script execution engine, an application program 
interface (105) for controlling an application program, an 
electronic program guide (EPG) infomriation acquiring 
means (106) for obtaining an electronic broadcast pro- 
gram guide, a user preference database DB (108) for 
analyzing preferences of the user from information pro- 
vided by the EPG infomriation acquiring means and the 
application program interface and storing analysis re- 
suits therein, an information providing engine (1 07) for 
providing information to the user according to informa- 
tion stored in the preference database, and a timer (112) 
for setting a time for providing information to the user by 
the Infomriation providing engine. 



EPC 
INFORMATION 
ACQUISITION 
MEANS 



APPLICATION 
PROGRAM 
INTERrACE 



INFORMATION 
PROVIDINQ 
ENCINE 











AGENT INTERFACE 



AGENT 
DISFVAV 
CONTT?OL 

MEANS 



SPEECH 
OUTPUT 
MEANS 



ACTION SCRIPT EXECUTIOH ENGINE 



RETntEVAL ENGINE 



PREFERENCES 
DATABASE 





Printed by Jouve, 7S001 PARIS (FH) 



INSOOCID: <EP 1189151A2J_s. 



1 



EP1 189151 A2 



2 



Description 

BACKGROUND OF THE INVENTION 

[0001] The present Invention relates to an agent In- 
terface device using an artificial agent. 
[0002] Japanese Laid-Open Patent Publication No. 
8-305531 disclosesaprlorarttechnlquebywhich aper- 
sonifled agent to be displayed on a personal computer 
of honne electronics is Interposed between a control ap- 
plication program of the home electronic apparatus and 
a user so that a user may control the electronic appara- 
tus by or via the agent. 

[0003] However, the prior art Involves the following 
problems. 

[0004] The technique disclosed in Japanese Laid- 
Open Patent Publication No. 8-305531 provides merely 
the same functions as those of a remote control of a TV 
set when viewing a TV program on a display of a per- 
sonal computer, so a user may loss a chance of viewing 
an interesting TV program if he or she does not positive- 
ly drive in time a television application program. 
[0005] The user must positively acquire TV program 
guides to know a schedule of broadcasting interesting 
TV programs. 

[0006] With the prior art, the user may not be aware 
if the application device was left as connected to the In- 
ternet for a long time. 

[0007] With the prior art, the user cannot know wheth- 
er resources become short when using a plurality of ap- 
plications. 

SUMMARY OF THE INVENTION 

[0008] The present invention was made to solve the 
above problems involved in the prior art device. 
[0009] Accordingly, the present invention is intended 
to provide an agent capable of teaching a user a TV pro- 
gram recommendable or suitable to the user preference 
a little while before the beginning of the program not to 
loss a chance of viewing it. 

[0010] The present Invention is also intended to pro- 
vide an agent capable of providing a guide and informa- 
tion of programs suited to the user preference. 
[0011] The present Invention Is further intended to 
provide an agent capable of advising the user to cut off 
the device connected to the Internet for a long time. 
[0012] The present invention is still further intended 
to provide an agent capable of informing the user of 
shortage of the system resource to prevent the system 
from unstably operating. 

[001 3] An object of the present invention is to provide 
an agent interface device for displaying an artificial 
agentfor providing infonnation by the agent's action and 
speech, comprising an agent interface having an agent 
display control means for controlling the display and ac- 
tion of the agent, a voice output means for generating a 
speech of the agent, and an input means for Input by a 



user, an action script execution engine for driving the 
agent interface into actual action according to an action 
script describing means to control the agent Interface, 
a retrieval engine for retrieving an adapted action script 

5 from an action script database and transferring the ac- 
tion script to the action script executing engine, an ap- 
plication program interface for controlling an application 
program, an electronic program guide infonnation ac- 
quiring means for acquiring an electronic broadcast pro- 

10 gram guide, a preference database for analyzing user 
preferences from infonnation provided by the electronic 
program guide infonnation acquiring means and the ap- 
plication program interface and storing analysis results 
therein, an infonnation providing engine for providing in- 

^5 formation to a user according to information stored in 
the preference database, and a timer for setting a time 
for providing infonnation to the user by the Infonnation 
providing engine. 

[0014] Another object of the present invention is to 

20 provide an agent Interface device, characterized in that 
it is further provided with means capable of deciding a 
user preferred television program from a user usage his- 
tory of a television application controllable by the appli- 
cation program interface according to infonnation from 

25 the electronic program guide information acquiring 
means and infonnation from the preference database, 
of displaying the agent at a specified time (n minute) 
before a start time of the user preferred TV program and, 
of causing the agent to have a conversation with the us- 

30 eraboutthe user preferred TV program by aspeech out- 
put through the voice outputting means. 
[0015] Another object of the present Invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of deciding a us- 

35 er preferred genre from a user usage history of a tele- 
vision application controllable by the application pro- 
gram interface according to infonnation from the elec- 
tronic program guide infonnation acquiring means and 
infonnation from the preference database, of displaying 

40 the agent at a specified time (n minutes) before a start 
time of the user preferred genre TV program, and of 
causing the agent to have a conversation with the user 
about the user preferred genre by a speech through the 
voice outputting means. 

45 [0016] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of displaying the 
agent immediately after the end of a TV program of a 
television application controllable by the application pro- 

50 gram interface and of causing the agent to have a con- 
versation with the user about the just viewed program. 
[0017] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is displayed immediately after ending a television 

55 application controllable by the application program in- 
terface and an Internet retrieval means for retrieving in- 
formation on the Internet is provided to retrieve informa- 
tion from a web site relevant to the just viewed program. 
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[0018] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is further provided with means capable of displaying the 
agent immediately after ending a video recording device 
controllable by the application program interface and of 
making conversation with the user about a just viewed 
video program. 

[0019] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is displayed immediately after ending a video re- 
cording device controllable by the application program 
interface and an internet retrieval means for retrieving 
infomriation on the Internet is further provided to retrieve 
infomnation from a web site relevant to a Just viewed vid- 
eo program. 

[0020] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is provided with means capable of displaying the agent 
immediately after ending an internet browser application 
program controllable by the application program inter- 
face and of causing the agent to make conversation with 
the user about a just browsed web site. 
[0021] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is displayed immediately after ending an internet 
browser application program controllable by the appli- 
cation program interface and an internet retrieval means 
for retrieving infomnation from a web site on an internet 
is provided to browse a web site relevant to a Just 
browsed site. 

[0022] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is capable of changing a theme of a conversation in ac- 
cordance with a keyword of a program displayed by a 
television application controllable by the application pro- 
gram interface. 

[0023] Another object of the present invention is to 
provide agent interface device, characterized in that it 
is capable of changing a theme of a conversation ac- 
cording to time interval infomiation obtainable from a us- 
age history of a television application controllable by the 
application program interface. 

[0024] Another object of the present invention is to 
provide agent interface device, characterized in that a 
table of frequently watched genre programs is prepared 
on the basis of a usage history of a television application 
controllable by the application program interface and 
displayed when driving a television application control- 
lable by the application program interface. 
[0025] Another object of the present invention is to 
provide agent interface device, characterized in that the 
agent is capable of informing the user of a communica- 
tion line stili connected to the Internet when n minutes 
elapsed after driving an internet browser application 
program controllable by the application program inter- 
face. 

[0026] Another object of the present invention is to 
provide agent interface device, characterized in that the 



agent is capable of informing the user of shortage of a 
system resource when a resource of the system runs 
short (it becomes n %). 

5 BRIEF DESCRIPTION OF DRAWINGS 

[0027] Fig. 1 is a construction view of an agent inter- 
face device according to the present invention. 
[0028] Fig. 2 is a module construction of an agent in- 
fo terface device according to the present invention. 

[0029] Fig. 3 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0030] Fig. 4 illustrates an exemplary scene of usage 
15 of an agent interface device according to an embodi- 
ment of the present invention. 

[0031] Fig. 5 illustrates an exemplary scene of an 
agent Interface device according to an embodiment of 
the present invention. 
20 [0032] Fig. 6 is a flowchart depicting the operation of 
an agent interface device according to an embodiment 
of the present invention. 

[0033] Fig. 7 is an exemplary display image of an 
agent of an agent interface device according to an em- 
25 bodiment of the present invention. 

[0034] Fig. 8 is a flowchart depicting the operation of 
an agent interface device according to an embodiment 
of the present invention. 

[0035] Fig. 9 is an exemplary display image of an 
30 agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0036] Fig. 10 illustrates an exemplary scene of usage 
of an agent interface device according to an embodi- 
ment of the present Invention. 
35 [0037] Fig. 11 illustrates an exemplary scene of usage 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0038] Fig. 12 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
40 ment of the present invention. 

[0039] Fig. 13 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0040] Fig. 14 is an exemplary display image of an 
45 agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0041] Fig. 15 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 
50 [0042] Fig. 16 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0043] Fig. 17 is a flowchart depicting the operation 
df an agent interface device according to an embodi- 
55 ment of the present invention. 

[0044] Fig. 18 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 



3 



EP 1 189151 A2J_> 



5 



EP 1 189 151 A2 



6 



[0045] Fig. 19 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

[0046] Fig. 20 is an exemplary display image of an 
agent of an agent interface device according to an em- 
bodiment of the present invention. 
[0047] Fig. 21 is a flowchart depicting the operation 
of an agent interface device according to an embodi- 
ment of the present invention. 

PREFERRED EI\^BODIMENT OF THE INVENTION 

[0048] A preferred embodiment of the present inven- 
tion will be described bellow with reference to the ac- 
companying drawings. 

[First embodiment] 

[0049] Fig. 1 Is a construction view of an agent inter- 
face device according to the present invention. 
[0050] A CPU 10, a memory 11 , a hard disk 12, a FD 
(Floppy Disk) drive portion 13, a communication portion 
14, an optical disk drive 15, a TV tuner board 1 6, a dis- 
play portion 20, a keyboard and mouse (pointing device) 
22 are connected to a bus 200. A microphone 1 8 and a 
speaker 1 9 are connected through a sound mixer 17 to 
the optical disk drive 15 and the TV tuner board 16 in 
order for the user to have a conversation with an agent 
of a current application. The communication device 14 
is connected to the Internet 23 to communicate with in- 
fo nnat ion sources on the Internet. 
[0051] Fig. 2 shows an example of a function module 
construction of an agent interface device according to 
the present Invention. The agent interface device com- 
prises an agent Interface 1 00, an action script execution 
engine 1 01 , a retrieval engine 1 02, an action scripts da- 
tabase 103, a dictionary 104, an application program in- 
terface 106, an electronic program guide (EPG) infor- 
mation acquisition means 106, an information providing 
engine 107, a user preference database DB108 and a 
timer 1 1 2 . An agent display control means 1 09 is a mod- 
ule used for presenting an artificial agent (a personified 
agent or an impersonated agent) on a display portion 
20 and controlling Its actions, a speech output means 
(voice output means) 110 is a module for providing the 
agent with voice and outputs a speech of the agent 
through the speaker 19. The speech may be a synthe- 
sized voice speech or recorded human voice speech. It 
Is not restricted to any of them. The input means 111 is 
a module for inputting a user's request to the agent by 
using a microphone 18, keyboard 21 or mouse (pointing 
device) 22. The input module 111 Includes a voice rec- 
ognition function for recognizing a speech inputted 
through the microphone 18, The agent interface 100 is 
composed of the agent display control means 109, the 
speech output means 110 and the input means 111. It 
serves as an Interface between the user and the appli- 
cation program Interface 105. This application program 



Interface 105 Is an interface that controls an application 
program and a control application program for a home 
electronic apparatus. The application program interface 
1 05 controls a variety of applications on a personal com- 

5 puter and control applications for home electronic appa- 
ratuses such as televisions, video recorders and optical 
disk record players. The action script database 1 03 con- 
tains action scripts each of which describes a procedure 
of realizing a particular function by co-operation of the 

10 agent interface 100 and the application program inter- 
face 105. An example of an action script is as follows: 
The agent informs the user of that a recommendable TV 
program suited to the user preference will start n min- 
utes later. 

15 [0052] The action script execution engine 101 analyz- 
es an actual action script and controls the agent inter- 
face 100 and the application program Interface 105 so 
that they may cooperatively execute the designated 
function. The retrieval engine 102 is a module that re- 

20 trieves an adaptive script from the action script database 
DB103 according to a retrieval request from the input 
means 111 and transfers the action script to the action 
script execution engine 1 01 . Since the retrieval request 
Inputted vocally through the Input means 111 Includes a 

25 natural language, the retrieval engine 102 refers to the 
dictionary 104 if necessary, retrieves an action script 
most suited to the request, for example, an action script 
for "driving a television application program and select- 
ing channel 2" from the action script database 103 and 

30 transfers it to the action script execution engine 101. The 
EPG information acquisition means 106 is used for ac- 
quiring an electronic television program guide (EPG). It 
may obtain the EPG, for example, from a web page for 
providing TV program infonnatlon by accessing It 

35 through the communication device 1 4 connected to the 
Internet 23 or may acquire the EPG from a data broad- 
cast by using the TV tuner board 16. The information 
providing engine 1 07 Is used for storing data In the pref- 
erence database DB108 and generating recommenda- 

40 tion to the user based on information accumulated in the 
preference database DB108. 

[0053] In case of TV programs, the infonnatlon pro- 
viding engine 1 07 obtains a TV state from the television 
application program interface which is one of the appll- 

45 cation program interface 105, refers to a program guide 
obtained from the EPG acquiring means 106 and 
records data of the program viewed by the user Into the 
user preference database DB108. Based on the user 
usage history In the user preference database DB108, 

50 the infonnatlon providing engine 1 07 refers to a table of 
programs obtained from the EPG acquiring means 106 
to be broadcast, acquires data of programs that the user 
may want to watch and transfers the data to the action 
script execution engine 101 . In this case, the infomia- 

55 tion-provlding engine may use the retrieval engine 102 
If necessary. 
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[Second Embodimentl 

[0054] Fig. 3 shows the first example of a screen Im- 
age of an agent, according to the second embodiment 
of the present Invention. Based on Information on pro- 
grams watched by the user and registered in the user 
preference database DB1 08, the agent interface device 
refers to a TV program guide acquired by the EPG in- 
formation acquisition means 106, finds therein a pro- 
gram to be preferably viewed by the user and timely 
transfers the selected program Information to the action 
script execution engine 101 . By the effect of the timer 
112, an agent is displayed on a display screen "n" min- 
utes before the beginning the program to inform the user 
that a recommended TV program will start n-mlnutes lat- 
er. For example, let assume that the user often viewed 
soccer game broadcasts on the TV set. In this case, the 
agent Interface device stores a user keyword "soccer 
game broadcast" in the user preference database DB, 
refers to a TV program guide obtained by the EPG ac- 
quiring means 106 to find therein a soccer game to be 
broadcast and obtain information including date, start 
time and channel of the broadcast. As soon as it reached 
"n" minutes before the beginning of the broadcast on the 
date, the agent Interface device displays on the TV dis- 
play an agent to inform the user that the broadcast of a 
soccer game will start soon. The time to provide infor- 
mation can be preset on the timer 112. Now let assume 
the time is set to 5 minutes. In this instance, the agent 
appears on the TV screen 5 minutes before the begin- 
ning of the broadcast and, for example, speaks to the 
user "your favorite soccer game will be soon on air. Do 
you want to see It?" and at the same time presents the 
same text in a window on the TV screen. When the user 
vocally answered "YES" (as shown In Fig. 4) or inputted 
a response "Yes" by using the keyboard 21 or mouse 
(pointing device) 22, the television application program 
is activated and indicates the channel 2 on the TV 
screen as shown in Fig. 5. 

[0055] Fig. 6 is a flowchart depicting the operation of 
the infonnation providing engine 1 07 of the agent inter- 
face device. When the application program started 
(Step S61 ), the infonnation providing engine 1 07 exam- 
ines the state of the television application program at 
specified inten/als (Step S62) and, if no TV program is 
displayed on the TV set (Step S63), proceeds to Step 
866. If the TV program is viewed in Step S63, the infor- 
mation providing engine 1 07 identifies the program that 
the user now enjoys, referring to an EPG (Electronic 
Program Guide) (Step S64) and, then, stores the TV 
program data into the user preference database DB1 08 
(Step 865). The EPG is obtained by the EPG infomna- 
tlon acquisition means 106. In practice, the EPG infor- 
mation is available from the Internet or data broadcast- 
ing by any of usual methods not defined herein. In Step 
S66, the infomnation providing engine 107 examines 
whether there Is a request to present recommendable 
programs. If no request is given (Step 867), the engine 



107 returns to Step S62. If any request was given (Step 
S67), the engine DB10B refers to the usage history con- 
tained in the user preference database DB1 08 to extract 
a keyword common to the programs that the user pre- 
5 f erred specially (Step S68). In this case, the engine 1 07 
may use the retrieval engine 102 if necessary. In Step 
S69. the engine 107 refers to the EPG of TV program 
to be broadcast and searches therein a program having 
the common keyword previously extracted from the da- 
te tabase DB108 or a similariy meaning keyword. In this 
case, the engine 1 07 may uses the retrieval engine 1 02 
if necessary. In Step S70, the engine 107 extracts the 
date and the start time of the program from the retrieved 
program information and examines by using the timer 
15 112 whether it is now n-mlnutes before the beginning of 
the program. If It Is not n-mlnutes before the start time, 
the engine 107 returns to Step S70. If It Is n-mlnutes 
before the start time, the engine 1 07 transfer the extract- 
ed program information to the action script execution en- 
20 gine 101 that is a origin of requesting for the information. 
The time-to-present may be optionally set by the user. 
The engine 107 returns to Step S62. The action script 
to be executed in this instance describes to present an 
agent who has a suitable conversation with the user In 
25 regard to the selected TV program. 

[0056] The above operation procedure causes the 
agent to appear on a TV screen "n" minutes before the 
start time of the program and have a conversation with 
the user in regard to the recommended TV program. 

30 

[Third Embodiment] 

[0057] In this embodiment, the genre-based recom- 
mendation Is used instead of the program-based rec- 

35 ommendation described for the second embodiment. 
The term "genre" means a category of TV programs ac- 
cording to their contents described in program Informa- 
tion acquired by the EPG acquiring means 1 06. Genres 
(categories) of TV programs are, for example, TV dra- 

40 ma, movie, sports, entertainment, hobbies/Living News, 
music, variety shows, social news and reports, educa- 
tion and animation/puppet shows, etc. The genres will 
not be further defined herein. 

[0058] The operation procedure similar to that of the 
45 second embodiment is executed to select genre-based 
recommendable programs, display an agent on a 
screen n-minutes before the start time of each of the 
genre-based recommended programs and make con- 
versation with the user in respect to the recommended 
50 genre-based program. 

[Fourth Embodiment] 

[0059] Fig. 7 shows the second example of a screen 
55 Image of an agent, according to the fourth embodiment 
of the present invention. The agent interface device of 
this embodiment refers to a TV program guide acquired 
by the EPG infomnation acquisition means 106 and ob- 
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tains therefrom Information on a TV program that the 
user is now enjoying. When the user ended the televi- 
sion application program, the agent interface device 
transfers information to the action script execution en- 
gine 101 and causes the same engine to present on a s 
TV screen an agent who may have a conversation with 
the user in respect to the TV program that the user 
viewed just before ending the Television application pro- 
gram. For example, let assume the user watched a soc- 
cer game broadcast and just finished the television ap- io 
plication program. In this case, the agent immediately 
appears on the TV screen and speaks "The live game 
in a stadium might be more exciting, don't you think so?" 
with the same text indicated in a window. 
[0060] Fig. 8 Is a flowchart depicting the operation of is 
the infomriatlon providing engine 1 07 of the agent inter- 
face device. When the program started (Step S81 ), the 
infomriatlon providing engine 107 examines the state of 
the television application program at specified intervals. 
(Step S82) and returns to Step S82 if no TV program is 20 
displayed on the TV set (Step S83). If the TV program 
is viewed in Step S83, the engine 107 refers to an EPG 
and identifies the program that the user now enjoys 
(Step S84). The engine 107 then extracts the TV pro- 
gram information from the EPG, which information in 
this instance may contain the program name and the 
genre name. In Step S86, the engine 107 examines 
whether the television application program ended, and, 
if not, returns to Step S84. If the program ended (Step 
S86), the engine 107 transfers the extracted action 30 
script (TV program information) to the action script ex- 
ecution engine 1 01 (Step S87) and then retums to Step 
S82. The action script describes that an agent is pre- 
sented to make conversation with the user in respect to 
the program infomnation. 35 
[0061] The above operation of the embodiment can 
realize such a particular effect that, as soon as the TV 
program ended, the agent appears on the TV screen to 
have a conversation with the user in respect to the just 
ended program. 40 

[Fifth Embodiment] 

[0062] Fig. 9 shows the third example of a screen im- 
age of an agent, according to the fifth embodiment of 45 
the present invention. This embodiment refers to a TV 
program guide acquired by the EPG information acqui- 
sition means 106 and obtains therefrom infomnation on 
a TV program that the user Is now viewing. When the 
user ended the television application program, the agent so 
interface device transfers information to the action script 
execution engine 101 to present an agent on a TV 
screen, then connects to the Internet 23 through the 
communication portion 14 and searches an information 
site relating to the program just viewed by the user. For 55 
example, it is assumed the user watched a soccer game 
broadcast and just ended the Television application pro- 
gram. In this instance, the agent immediately appears 



on the TV screen and speaks "How about searching an 
Internet web page relating to <your favorite soccer>?" 
with the same text Indicated in a window. When the user 
vocally answered "Yes" or inputted the same response 
by using the keyboard 21 or the mouse (pointing device) 
22, the agent interface device drives an applteation pro- 
gram to carry out browsing a "soccer" web site on the 
Internet. 

[0063] Fig. 12 Is a flowchart depicting the operation 
of the information providing engine 1 07 of the agent in- 
terface device. When the program started (Step SI 21), 
the information providing engine 107 examines the cur- 
rent state of the television application program at spec- 
ified intervals (Step SI 22) and retums to Step SI 22 if it 
judged no TV program being viewed on the TV set (Step 
S123). Having judged aTV program being viewed (Step 
SI 23), the engine 107 refers to an EPG and identifies 
the program that the user now enjoys (Step SI 24). The 
engine 107 then extracts a keyword, e.g., the TV pro- 
gram title (Step SI 25). In Step S126, the engine 107 
examines whether the user ended the television appli- 
cation program. If the television application program is 
not ended, the engine 107 retums to Step SI 24. When 
the television application program ended (at Step 
SI 26), the engine 107 transfers an action script to the 
extracted keyword to the action script execution engine 
101 (Step S127) and then returns to Step S122. The 
action script describes that an Internet browser applica- 
tion program shall be driven to search for the keyword 
and browse relevant web sites. 

[0064] The above operation of this embodiment real- 
izes such a particular effect that web site pages relating 
to the TV program just watched by the user is explored 
and browsed as soon as the TV program ended. 

[Sixth embodiment] 

[0065] This embodiment is analogous to the fourth 
embodiment but deals with video programs recorded on 
the recording devices (media) such as a video recording 
tape, CD-RW, DVD-RW, MO and hard disk, while the 
latter deals with TV programs viewable by the television 
application. Therefore, the agent interface device In this 
embodiment recognizes the end of a video program (i. 
e., a recording device) and transfers information on the 
video program to the action script execution engine 101. 
[0066] Based on the video program information, this 
embodiment conducts the same procedure as that of the 
fourth embodiment to display an agent immediately after 
the end of the video program and have a conversation 
with the user in regard to the just watched program. 

[Seventh Embodiment] 

[0067] This embodiment is analogous to the fifth em- 
bodiment but deals with video programs recorded on the 
recording devices (media) such as a video recording 
tape, CD-RW, DVD-RW, MO and hard disk, while the 
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latter deals with TV programs viewable by the television 
application. Therefore, the agent interface device in this 
embodiment recognizes the end of a video program (i. 
e., a recording device) and transfers information on the 
video program to the action script execution engine 101. 5 
[0068] Based on this infonnation, this embodiment 
displays an agent immediately after the end of the video 
program and starts searching a web site relating to the 
just watched program. 

10 

[Eighth Embodiment] 

[0069] In this embodiment, the television application 
described in the fourth embodiment is replaced by an 
Internet browser application program and information of ^5 
an browsed web site is transferred to the script execu- 
tion engine 1 01 after recognition of finishing the Internet 
browser application program. 

[0070] The operation of the embodiment realizes 
such a particular effect that an agent appears on a dis- 20 
play screen immediately after the end of the Internet 
browser application program to have a conversation 
with the user in regard to the just viewed web site. 

[Ninth Embodiment] 25 

[0071] In this embodiment, the television application 
described in the fifth embodiment is replaced by an In- 
ternet browser application program and infomiation of 
an browsed web site is transferred to the script execu- 30 
tion engine 1 01 after recognition of finishing the Internet 
browser application program. 

[0072] The operation of the embodiment realizes that 
an agent appears on a display screen immediately after 
the end of the Internet browser application program and 35 
starts browsing another web site retatingthe just viewed 
web site In the Internet. 

[Tenth Embodiment] 

40 

[0073] In this embodiment, an agent can change the 
theme of a conversation with the user depending on a 
keyword of a TV program being viewed by the user. For 
example, when the user did an incorrect operation with 
watching no TV program, an agent appears on a display ^5 
screen and outputs a message "I cannot accept your 
operation" in both in the speech mode and the text 
mode. For example, if a keyword "soccer" is extracted 
while watching the soccer game broadcast, the agent 
appears and outputs a spoken message and a same so 
text message regarding the extracted keyword, e.g., "I 
cannot accept your request. You may receive a Red card 
if you should repeat the incorrect operation so may 
times". 

[0074] Fig. 13 is a flowchart depicting the operation 55 
of the information providing engine 107 of the tenth em- 
bodiment. Once the program started in Step S131 , the 
infonnation providing engine 107 examines a cun-ent 



state of the television application program at specified 
time intervals (Step SI 32). If no TV program Is viewed 
(Step S1 33), the engine 1 07 returns to Step SI 32. When 
a TV program is watched (Step S133), the engine 107 
refers to the EPG and extracts therefrom a keyword (e. 
g., title) of the program being viewed by the user (Step 
SI 34). The engine 107 transfers the obtained keyword 
to the action script execution engine 101 and then re- 
turns to Step SI 32. The action script may be selected 
depending on the keyword and the conversation script 
may be decided depending on the keyword. 
[0075] The above operation of the embodiment pro- 
vides such a particular effect that a conversation theme 
can be changed depending on a detected keyword. 

[Eleventh Embodiment] 

[0076] Fig. 14 shows the fourth example of a display 
image of an agent, according to the eleventh embodi- 
ment of the present invention. When one of programs 
frequently viewed by the user and registered in the user 
preference database DB108 has not been watched by 
the user within the time interval of n days after the last 
watching date, the embodiment transfers the informa- 
tion to the action script execution engine 1 01 that in turn 
displays the agent on a display screen to execute the 
conversation script regarding the fact that the user has 
not watched the favorite program. For example, when 
the time interval from the last watching a soccer game 
broadcast exceeds 30 days, the agent is displayed to 
output a message "Why didn't you watch recent soccer 
game broadcasts? Your favorite team has lost 
games? ". This message can be outputted in the speech 
mode and the text mode at a time. 
[0077] Fig. 15 is a flowchart depicting the operation 
of the infomnation providing engine 107 in the eleventh 
embodiment. Once the program started in Step S151, 
the inf omnation providing engine 1 07 examines program 
information registered in the preference database 
DB108 at specified time intervals (Step SI 52). In Step 
SI S3, the engine 1 07 examines whether any one of the 
programs registered in the user preference database 
DB1 08 has an interval of more than n days from the last 
watching date. If not, the engine 107 returns to Step 
S152. When there is found a program having the non- 
watched period exceeding n days , the engine 1 07 
transfers the detected program information to the action 
script execution engine 101 and then returns to Step 
SI 52. The action script to be executed is decided in ac- 
cord with the non-watched period. The conversation 
content may be changed depending on the non- 
watched period (time interval). 

[0078] As the result of the above operation of the em- 
bodiment, a theme of the conversation can be selected 
In accord with the non-watched period. 
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[Twelfth Embodiment] 

[0079] Fig. 16 shows the fifth exampleof a display Im- 
age of an agent, according to the twelfth embodiment of 
the present Invention, The agent interface device iden- 
tifies a user preferred program genre by calculating a 
sum of viewing frequencies for each genre from usage 
history infonnation stored in the preference database 
DB1 08, recognizes the time of starting the television ap- 
plication program by the timer 23, selects the preferred 
genre programs being cun'ently broadcast or to be 
broadcast after now from a TV program guide obtained 
by the EPG acquisition means 1 06 and displays a table 
of genre-based preferred programs on the display 
screen. For example, if the user has frequently viewed 
broadcasts of soccer games, sports is identified as the 
preferred genre and a table of programs of the sport 
genre is displayed. 

[0080] Fig. 17 is a flowchart depicting the operation 
of the Information providing engine 107 of the twelfth 
embodiment. Once the program started in Step S171, 
the infonnation providing engine 107 extracts the view- 
ing frequency data for each genre from the usage history 
information stored In the user preference database 
DB108 at specified time intervals (Step SI 72). Namely, 
the engine 81 07 identifies the genre of programs having 
the highest viewing frequency (Step S1 73). The engine 
107 examines whether the television application pro- 
gram is driven or not (Step S 1 74). If not, the engine 1 07 
returns to Step SI 72. If the television application pro- 
gram was driven, the engine 107 recognizes, from the 
timer 112, the time of starting the television application 
program. The engine 107 obtains a table of programs 
being currently broadcast or to be broadcast from TV 
program guides acquired by the EPG acquisition means 
1 06 (Step SI 76) and extracts from the above table only 
programs of the genre decided in Step SI 73. Finally, the 
engine 1 07 displays a table of selected programs on the 
display screen (Step S178) and then returns to Step 
S172. 

[0081] As the result of the above operation of the em- 
bodiment, a table of preferred genre programs can be 
prepared based on the usage history and displayed on 
the display screen immediately after starting the televi- 
sion application program. 

[Thirteenth Embodiment] 

[0082] Fig. 18 shows the sixth example of a display 
image of an agent, according to of the thirteenth embod- 
iment of the present invention. Once the user activated 
an Internet browser application program to browse web 
sites, this embodiment monitors the duration of connec- 
tion of the user's communication line to the Internet by 
the timer 11 2 and, when n minutes elapsed, displays the 
agent on the display screen to infomi the user that the 
communication line is still connected to the Internet. 
For example, when 120 minutes elapsed after starting 



the Internet browsing application and connecting the 
communication line to the Internet, the agent is dis- 
played on the display screen to output a warning mes- 
sage "Your line is always busy. The connection is still 
5 live". This message may be outputted vocally and visu- 
ally at a time. 

[0083] Fig. 19 is a flowchart depicting the operation 
of an agent interface device according to the thirteenth 
embodiment of the present Invention. Once the program 

10 started in Step S191 , the agent interface device exam- 
ines whether the Internet browser application program 
is driven (Step SI 92). If not, the device returns to Step 
SI 92. If the program was driven, the device examines 
whether the communication line Is connected to the In- 

15 ternet (Step SI 93). If not, the device returns to Step 
SI 92. When the communication line is connected to the 
Internet, the device counts the connection period by the 
timer 112. The device examines whetherthe connection 
period reaches n minutes (Step SI 95). If not, the device 

20 returns to Step SI 92. If n minutes elapsed, the device 
transfers an action script descriptive of "the communi- 
cation line being left as connected to the Internet" to the 
action script execution engine 1 01 . The time setting can 
be freely changed by the user. The device then returns 

25 to Steps 192. 

[0084] As the result of the above operation of the em- 
bodiment, the agent can Inform the user of the commu- 
nication line being left as connected to the Internet when 
n minutes elapsed after driving the internet browser ap- 

30 plication program. 

[Fourteenth Embodiment] 

[0085] Fig. 20 shows the seventh example of a display 

35 image of an agent, according to the fourteenth embod- 
iment of the present invention. When the user activated 
a plurality of application programs and the remaining ca- 
pacity of the system resource was reduced to n%, this 
embodiment displays the agent on the display screen to 

40 inform the user that the operation will be unstable due 
to shortage of the system resource. For example, when 
the remaining capacity of the system resource was re- 
duced to 20%, the agent appears on the display screen 
and outputs a waming message "The system is in an 

45 unstable state. Please finish any currently unused pro- 
gram". This message can be outputted in both the 
speech mode and the text mode at a time. 
[0086] Fig. 21 Is a flowchart depicting the operation 
of an agent interface device according to the fourteenth 

50 embodiment of the present invention. When the pro- 
gram started in Step S211, the agent interface device 
examines the remaining capacity of the system re- 
source (Step S212) and decides whetherthe remaining 
resource capacity is no more than h %. In Step S213, if 

55 it is more than n %, the device returns to Step S212. If 
it is no more than n %, the device transfers action script 
describing "shortage of the system resource" to the ac- 
tion script execution engine 101 (Step S214). 
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[0087] As the result of the above operation of the em- 
bodiment, the agent can infomri the user of the shortage 
of the systenn resource when the remaining resource ca- 
pacity is reduced to n %. 

[0088] It should be understood that the foregoing re- 
lates to only preferred embodiments of the invention, 
and that the invention is not limited to the specific em- 
bodiments thereof except as defined in the appended 
claims. 

[0089] The agent interface device according to the 
present invention can offer the following advantages: 
[0090] The user can operate a variety of applications 
through interactions with the personified agent inter- 
face. Namely, the agent interface device may serves as 
a unified interface for a variety of applications. 
[0091] The user may not loss a chance of viewing a 
favorite television program because the agent previous- 
ly infomns him/her that the program will be broadcasted. 
[0092] The user can enjoy a conversation with the 
agent who can speak about the television program the 
user has just viewed. 

[0093] Detailed data related to the television program 
that the user just has viewed can be obtained from web 
sites by browsing, which is realized through the Interac- 
tion with the agent interface. 

[0094] The agent interface can prepares a table of 
preferred genre programs based on the usage history 
data stored In the user preference database and pre- 
sented it on the display screen, wherefrom the user can 
know new interesting programs that he/she has not 
viewed. 

[0095] When the communication line was left as con- 
nected to the Intemet for a specified time, the agent in- 
terface Informs the user of the connected state of the 
communication line. The user can cut off the communi- 
cation line in due time. 

[0096] The agent interface informs the user that the 
remaining capacity of the system resource may be short 
for the current applications. This enables the user to 
take suitable measures to avoid the unstable operation 
of the system. 



Claims 

1 . An agent interface device for displaying an artificial 
agent for providing information by the agent's action 
and speech, comprising: 

an agent interface (100) having an agent dis- 
play control means (109) for controlling the dis- 
play and action of the agent, a voice output 
means (110) for generating a speech of the 
agent, and an input means (111) for input by a 
user, 

an action script execution engine (1 01 ) for driv- 
ing the agent interface into actual action ac- 
cording to an action script describing means to 



control the agent interface, 
a retrieval engine (102) for retrieving an adapt- 
ed action script from an action script database 
and transferring the action script to the action 

5 script executing engine, 

an application program interface (1 05) for con- 
trolling an application program, 
an electronic program guide information acquir- 
ing means (106) for acquiring an electronic 

10 broadcast program guide, 

a preference database (1 08) for analyzing user 
preferences from information provided by the 
electronic program guide information acquiring 
means and the application program interface 

15 and storing analysis results therein, 

an infomnation providing engine (107) for pro- 
viding information to a user according to infor- 
mation stored in the preference database, and 
a timer (1 1 2) for setting a time for providing in- 

20 formation to the user by the information provid- 

ing engine. 

2. An agent interface device as defined in claim 1 , 
characterized in that it is further provided with 

25 means capable of deciding a user preferred televi- 
sion program from a user usage history of a televi- 
sion application controllable by the application pro- 
gram interface (105) according to information from 
the electronic program guide infonnation acquiring 

30 means (106) and information from the preference 
database (1 08), of displaying the agent at a speci- 
fied time (n minute) before a start time of the user 
preferred TV program and, of causing the agent to 
have a conversation with the user about the user 

35 preferred TV program by a speech output through 
the voice outputting means. 

3. An agent interface device as defined in claim 1, 
characterized in that it is further provided with 

40 means capable of deciding a user preferred genre 
from a user usage history of a television application 
controllable by the application program interface 
(105) according to information from the electronic 
program guide Infomnation acquiring means (106) 

45 and infomnation from the preference database 
(108), of displaying the agent at a specified time (n 
minutes) before a start time of the user preferred 
genre TV program, and of causing the agent to have 
a conversation with the user about the user pre- 

50 ferred genre by a speech through the voice output- 
ting means (110). 

4. An agent interface device as defined in claim 1, 
characterized in that it is further provided with 

55 means capable of displaying the agent immediately 
after the end of a TV program of a television appli- 
cation controllable by the application program inter- 
face (1 05) and of causing the agent to have a con- 



9 



WSDOCID: <EP 1189161A2J_> 



17 



EP 1 189 151 A2 



18 



versation with the user about the just viewed pro- 
gram. 

5. An agent interface device as defined in claim 4, 
characterized in that the agent is displayed imme- s 
diately after ending a television appiication control- 
lable by the application program interface (1 05) and 

an internet retrieval means for retrieving infomnation 
on the Internet is provided to retrieve Information 
from a web site relevant to the just viewed program. io 

6. An agent interface device as defined in claim 4, 
characterized in that it is further provided with 
means capable of displaying the agent Immediately 
after ending a video recording device controllable is 
by the application program interface (105) and of 
making conversation with the user about a just 
viewed video program. 

7. An agent interface device as defined in claim 5, 20 
characterized in that the agent is displayed imme- 
diately after ending a video recording device con- 
trollable by the application program interface (105) 
and an intemet retrieval means for retrieving infor- 
mation on the Internet is further provided to retrieve 25 
infomnation from a web site relevant to a just viewed 
video program. 

8. An agent interface device as a defined In claim 4, 
characterized in that it is provided with means ca- 30 
pable of displaying the agent immediately after end- 
ing an internet browser application program control- 
lable by the application program Interface (1 05) and 

of causing the agent to make conversation with the 
user about a just browsed web site. 35 

9. An agent interface device as defined in claim 5, 
characterized in that the agent is displayed imme- 
diately after ending an internet browser application 
program controllable by the application program in- 40 
terface (1 05) and an intemet retrieval means for re- 
trieving infomriation from a web site on an internet 

is provided to browse a web site relevant to a just 
browsed site. 

45 

10. An agent interface device as defined in claim 1, 
characterized in that it is capable of changing a 
theme of a conversation in accordance with a key- 
word of a program displayed by a television appli- 
cation controllable by the application program inter- so 
face (105). 

11. An agent interface device as defined in claim 1, 
characterized in that it is capable of changing a 
theme of a conversation according to time interval 55 
information obtainable from a usage history of a tel- 
evision application controllable by the application 
program interface (105). 



12. An agent interface device as defined in claim 1, 
characterized in that a table of frequently watched 
genre programs is prepared on the basis of a usage 
history of a television application controllable by the 
appiication program interface (105) and displayed 
when driving a television application controllable by 
the appiication program interface. 

13. An agent Interface device as defined in claim 1, 
characterized in that the agent is capable of in- 
fomrjing the user of a communication line still con- 
nected to the Internet when n minutes elapsed after 
driving an intemet browser application program 
controllable by the application program interface 
(105). 

14. An agent interface device as defined in claim 1, 
characterized In that the agent is capable of in- 
forming the user of shortage of a system resource 
when a resource of the system runs short (it be- 
comes n %). 



10 

3NSDOCID: <EP 1189151A2_I_> 



EP1 189151 A2 




O 
O 
CVi 



GO 





in 




CO 
CM 





CM 
CM 




CM 




O 




CO 



CM 





INSDOCID: <EP 1189151A2J_> 



11 



EP 1 189 151 A2 



CSJ 

d 

Li- 




3NSEXX;iD: <EP 1189151A2_L> 



12 



EP 1 189 151 A2 



FIG.3 



BROADCASTING A 
SOCCER GAME 

OSAKA VS. NARA 
CHANNEL: 2CH 
KICK-OFF TIME: 
PM7:00 




YOUR FAVORITE SOCCER 
GAM WILL SOON START. 
DO YOU WANT TO SEE IT? 
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A WEB PAGE RELATING TO 
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ON THE INTERNET? 
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